Decoder apparatus and method for smoothing artifacts created during error concealment

ABSTRACT

Errors in coded macroblocks are concealed during decoding by way of an error concealment stage present in a decoder. Error-concealed macroblocks produced by the error concealment stage undergo deblocking filtering by a deblocking filter before being output by the decoder to avoid the spread of erroneous pixel values. The error concealment stage controls the deblocking filter in accordance with the error concealment technique to vary the strength of the deblocking filter to force maximum strength on transitions artificially created by the recovery of lost macroblocks.

CROSS REFERENCE TO RELATED APPLICATION

This application claims priority under 35 U.S.C. 119(e) to U.S.Provisional Patent Application Ser. No 60/439,312, filed Jan. 10, 2003,the teachings of which are incorporated herein.

TECHNICAL FIELD

This invention relates to a video decoder that performs errorconcealment to mitigate errors caused by missing or corrupted data.

BACKGROUND ART

In many instances, video streams undergo compression (coding) tofacilitate storage and transmission. Not infrequently, such coded videostreams incur data losses or become corrupted during transmissionbecause of channel errors and/or network congestion. Upon decoding, theloss/corruption of data manifests itself as missing pixel values. Toreduce artifacts attributable to such missing/corrupted pixel values, adecoder will “conceal” such missing/corrupted pixel values by estimatingthe values from other macroblocks in the same image or from anotherimage. The term conceal is a somewhat of a misnomer because the decoderdoes not actually hide missing or corrupted pixel values.

Despite the importance of error concealment, most decoders usuallyimplement only the simplest and fastest concealment algorithms forreal-time applications. For most real-time applications, there exist twodifferent approaches for achieving error concealment. One approachproposes the replacement of the missing macroblock by copying one of itscorrectly decoded neighbors. This approach finds application on lowquality systems given that the blocking artifacts appearing on thereconstructed image are highly visible. The second approach attempts tosmooth the blocking artifacts by interpolating the content of themissing macroblock based on the pixel values on the border of correctlydecoded neighboring macroblocks. Two different schemes fit into thislatter category: (1) replacement of all the pixels within amacroblock/block with a common mean value; and (2) replacement of eachpixel value by means of a weighted prediction based on the pixeldistance to the macroblock/block boundaries. With no criteria todistinguish between flat and contoured regions, this concealmentapproach tends to blur the reconstructed image creating the oppositeartifact.

Thus, there is need for a concealment approach that achieves simplicityand high performance when reducing the blocking artifacts created by thederivation process of the missing/corrupted pixel values.

BRIEF SUMMARY OF THE INVENTION

Briefly, in accordance with a preferred embodiment of the presentprinciples, a video decoder compliant with the ISO/ITU H.264 videocompression standard includes an error concealment stage for concealingerrors in decoded macroblocks that have missing/corrupted pixel values.The error concealment stage performs such error concealment byestimating the missing/corrupted pixel values from previouslytransmitted macroblocks that are error free. The macroblocks produced bythe error concealment stage are input to a deblocking filter in thedecoder that deblocks transitions artificially created by the inaccuracyof the error concealment process. In other words, the error concealmentstage performs error concealment in advance of filtering by thedeblocking filter. Advantages of such approach are twofold. First, byusing the deblocking filter to enhance the results of the errorconcealment method, high quality can be achieved with low complexityrequirements. Secondly, error correction prior deblocking avoids thespread erroneous pixel values when attempting to smooth transitionsbetween erroneous and correctly decoded blocks.

In accordance with another aspect of the present principles, the errorconcealment stage varies the parameters of the deblocking filter. Inparticular, the error concealment stage varies the parameters of thedeblocking filter to force maximum filter strength on the transitionsartificially created by the recovery of lost macroblocks.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 depicts a block schematic diagram of a decoder that provideserror concealment in accordance with the present principles; and

FIG. 2 depicts in flow chart form the process by which the decoder ofFIG. 1 operates to accomplish error concealment.

DETAILED DESCRIPTION

FIG. 1 illustrates a block schematic diagram of a video decoder 10compliant with the ISO/ITU H.264 compression standard for accomplishingerror concealment in accordance with the present principles. The decoder10 includes an entropy decoding stage 12 that receives an input bitstream representative of a video signal compressed (encoded) by anupstream encoder (not shown) in accordance with the H.264 compressionstandard. The entropy decoding stage 12 decodes the input stream toyield: (a) transformed coefficients, (b) motion vectors and referenceframe indices, and (c) control data A scaling/inverse transformationstage 14 receives the transformed coefficients for inversetransformation and scaling to re-create the prediction error. Theprediction error reflects the difference between the original image atthe encoder and the estimated image the decoder can obtain based onpreviously transmitted data. The prediction error produced by thescaling/inverse transformation stage 14 passes to a summing block 18 forsumming with the estimated image obtained either by inter orintra-prediction.

For an input macroblock encoded in inter prediction mode, the motioncompensation stage 16 serves to produce the estimated image, from inputinformation including the motion vectors and the reference frame indicessent in the input bit-stream and corresponding reference framespreviously stored in the decoder buffer. The output from the motioncompensation stage 16 passes to the summing block 18 for summing withthe error prediction produced by the scaling/inverse transform stage 14to produce the reconstructed image. Each macroblock in the reconstructedimage output from the summing block 18 passes to an error concealmentstage 20, which detects whether the macroblock has missing or corruptedpixel values. If so, the error concealment stage 20 will substituteestimated pixel values in place of those that are lost or corrupted. Theerror-concealed macroblock output by the error concealment stage 20undergoes deblocking at deblocking filter 22. In accordance with thepresent principles, the deblocking filter 22 has adjustable parametersto allow varying of the strength of the filtering performed on theconcealed image. The deblocking filter 22 produces the output image ofthe decoder 10. At this point, those images marked as reference imagesin the bit-stream are stored in the reference frame buffer to serve asone of the inputs to the motion compensation block 16.

For input macroblocks encoded in intra prediction mode, theintra-prediction stage 24 will produce the estimated image in accordancewith the intra-prediction modes sent on the coded input bit-stream. Theestimated image produced by the intra-prediction stage 24 passes to thesumming block 18 for summing with the error prediction produced by thescaling/inverse transform stage 14 to produce the reconstructed image.Each inter-predicted macroblock output by the summing block 18, likeeach intra predicted macroblock output by the summing block, undergoeserror concealment at the error concealment stage 20 and then deblockingby the deblocking filter 22.

FIG. 2 illustrates in flow chart form the steps undertaken by the errorconcealment stage 20 within the decoder 10 of FIG. 1 to accomplish errorconcealment and to adjust the parameters of the deblocking filter 22 toachieve maximum filtering on the transitions resulting from errorconcealment. The error concealment stage 20 initiates error concealmentduring step 100 of FIG. 2 by performing error detection on eachsuccessive input macroblock received from the summing block 18 ofFIG. 1. Upon detecting no error during step 120, then the errorconcealment stage ends the error concealment process (step 125 of FIG.2) and outputs the received macroblock to the deblocking filter to 22with no corrections. In the absence of making any error concealment tothe received macroblock, the error concealment stage makes no adjustmentto the parameters of the deblocking filter 22 of FIG. 1.

Should an error exist, as determined during step 120, the errorconcealment stage 20 of FIG. 1 makes a determination during step 140 ofFIG. 2 whether the macroblock received from the summing block 18 of FIG.1 has been intra-coded. An intra-coded block having errors undergoesspatial error concealment during step 160, whereas an inter-coded blockundergoes temporal concealment during step 180.

There exist various techniques for spatial error concealment, including:

-   -   Block copy (BC)

With this approach, the replacement of a missing/corrupted macroblock isobtained from one of its correctly decoded neighbors.

-   -   Pixel Domain Interpolation (PDI):

The missing/corrupted macroblock data is interpolated from the pixelvalues at the border of the correctly decoded neighbors. Two differentapproaches exist for accomplishing PDL For example, all the pixelswithin a macroblock can be interpolated to a common mean value.Alternatively, each pixel value is obtained by means of a weightedprediction based on the pixel distance to the macroblock boundaries.

-   -   Multi-Directional Interpolation (MDI)

The multi-directional interpolation technique constitutes an improvedversion of the PDI technique because the MDI technique providesinterpolation along the edge directions. Accomplishing MDI requiresestimating the directions of the main contours in the neighborhood ofthe missing/corrupted pixel value prior to directional interpolation.

-   -   Maximally Smooth Recovery (MSR):

In the Discrete Cosine Transformation (DCT) domain, low frequencycomponents are used for error concealment to provide a smooth connectionwith the adjacent pixels. When data-partitioning encoding is used, theMSR technique exploits the correctly received DCT coefficients insteadof discarding all the data within the corrupted macroblock/block.

-   -   Projection On Convex Sets (POCS):

In accordance with this technique, adaptive filtering is performed inthe Fast Fourier Transform (FFT) domain, based on the classification ofa larger region surrounding the macroblock with missing/corrupted pixelvalues. Such adaptive filtering includes the application of low-passfiltering on smooth regions while applying an edge filter on sharpregions. This procedure includes a filtering iteration and several apriori constraints will apply to the treated image.

In addition to the foregoing techniques, spatial error concealment canbe advantageously achieved the following manner. For each identifiedmacroblock, at least one intra-prediction mode is derived fromneighboring macroblocks. When the image is coded in accordance with theISO/ITU H.264 video compression standard, two intra-coding types areavailable for the coding of each macroblock: (1) for an Intra_(—)16×16type, a single intra prediction mode is derived for the wholemacroblock; (2) for an Intra_(—)4×4 type, an intra prediction mode isderived for each sub-macroblock of 4×4 pixels within the macroblock. (Inthis case, there are sixteen intra prediction modes per codedmacroblock.). The derived intra-prediction modes are then applied togenerate the missing pixel values. The process by which the derivedintra prediction modes are applied to estimate missing or corruptedpixel values corresponds to the derivation process employed duringdecoding to estimate (predict) the non-coded values to reduce the codingeffort. In other words, the present technique utilizes the intraprediction mode information normally used in coding for spatial errorconcealment purposes. When the coded data referring to a particularmacroblock is lost or corrupted, the intra prediction modes derived fromneighboring macroblocks can provide important information about which isthe best interpolation direction for spatial error concealment. Usingsuch intra prediction modes for spatial error concealment yieldssignificantly better performance than the classical spatial errorconcealment techniques with similar complexity.

In contrast to spatial error concealment, temporal concealment attemptsthe recovery of the coded motion information, namely the referencepicture indices and the motion vectors, to estimate the missing pixelvalues from a previously transmitted macroblock. Recovery of theprediction error from the same macroblock is unfeasible since thisinformation is coded without redundancy. Unlike spatial concealment,fundamentals of temporal concealment are almost the same in most of thepublished algorithms. Because it is computationally expensive to searchfor a missing motion vector of a missing macroblock in one or morereference frames, typically only a limited set of candidates isconsidered. Possible motion vectors for consideration include:

-   -   Zero motion: assumes that the lost block hasn't changed its        location between two consecutive frames and performs the        temporal concealment by simply copying the collocated block on        the previous frame.    -   Global motion: assumes that the lost block undergoes a global        motion, and that can be correctly approximated in most of the        cases by estimating the camera motion parameters.    -   Local motion: assumes that the motion of spatially neighboring        blocks is highly correlated and hence the motion of a missing        block can be recovered from the local motion information        available on its neighborhood.

Following either spatial error concealment during step 160 or temporalconcealment during step 180, the error concealment stage 20 of FIG. 1adjusts the parameters of the deblocking filter 22 of FIG. 1 to forcemaximum strength filtering on the transitions artificially created bythe recovery of lost macroblocks. As defined by the H.264 standard, theintensity of the deblocking filter 22 adapts to the characteristics ofeach edge between blocks of 4×4 pixels. Adaptation is done depending onthe following parameters:

-   -   The boundary strength value (Bs), computed at the decoder 10.    -   The Quantization Parameter (QP) average computed at the decoder        10 between any pair of blocks affected by the deblocking filter        22.    -   The filter offsets A and B transmitted in the slice header.

The boundary strength value, ranging from 0 to 4, designates thestrength of the filtering that applies to the edge between two 4×4 pixelblocks. When Bs=0, the edge remains unfiltered. When Bs=4, the edge issmoothed by the strongest filter strength. The other parameters, namelythe QP average and the filter offsets A and B, are jointly used todetermine the thresholds that differentiate real contours fromartificial transitions. High values of these parameters increase thenumber of filtered transitions.

In accordance with the present principles, the chosen error concealmentalgorithm will vary the boundary strength value, or any of the inputparameters that, after computation, return the desired boundary strengthvalue. Alteration of the boundary strength value can be done on theedges between pairs of concealed blocks and/or on the edges between theconcealed blocks and correctly received ones. Ultimately whether it isappropriate or not to increase the strength of the deblocking filter andby what value depends on the particular technique chosen for errorconcealment.

In an illustrative embodiment, the maximal boundary strength value of(4) was chosen on the edges between pairs of blocks concealedindependently. The particular error concealment technique could alsochange the value of the QP average between any pair of blocks and/or theoffset values transmitted on the header of the corrupted slice. Changingthe value of the QP average will increase the number of filteredtransitions. In the illustrative embodiment, all parameters are forcedto their maximal value, i.e. 51 for the QP average and 6 for the offsetsA and B.

The foregoing describes a technique for accomplishing error concealmentin a H.264 compliant decoder and for altering the strength of deblockingin accordance with the type of error concealment performed.

1. In combination with an ISO/ITU H.264 compliant video decodercontaining a deblocking filter, an error concealment stage that receivesdecoded macroblocks for concealing errors in macroblocks havingmissing/corrupted data by estimating their pixel values from previouslytransmitted macroblocks to yield error concealed macroblocks for inputto the deblocking filter which by deblocking the error concealedmacroblocks avoids the spread of erroneous pixel values.
 2. The decoderaccording to claim 1 wherein the error concealment stage varies thestrength of the deblocking performed by the deblocking filter inaccordance with error concealment.
 3. The decoder according to claim 2wherein the error concealment stage modifies the strength of thedeblocking filter by modifying a boundary strength value on transitionsbetween concealed macroblocks and error-free (correctly received)macroblocks.
 4. The decoder according to claim 2 wherein the errorconcealment stage modifies the strength of the deblocking filter bymodifying a boundary strength value on transitions between pairs ofconcealed macroblocks.
 5. The decoder according to claim 2 wherein theerror concealment stage modifies a Quantization Parameter (QP) averagefor the deblocking filter between concealed macroblocks and correctlyreceived macroblocks.
 6. The decoder according to claim 2 wherein theerror concealment stage modifies a Quantization Parameter (QP) averagefor the deblocking filter between pairs of concealed macroblocks.
 7. Thedecoder according to claim 3 wherein the error concealment stagemodifies a Quantization Parameter (QP) average for the deblocking filterbetween concealed macroblocks and correctly received macroblocks.
 8. Thedecoder according to claim 4 wherein the error concealment stagemodifies a Quantization Parameter (QP) average for the deblocking filterbetween pairs of concealed macroblocks.
 9. The decoder according toclaim 2 wherein the error concealment stages varies each of a pair ofoffset values A and B for the deblocking filter.
 10. The decoderaccording to claim 9 wherein the error concealment stage modifies thestrength of the deblocking filter by modifying a boundary strength valueon transitions between concealed macroblocks and error-free (correctlyreceived) macroblocks.
 11. The decoder according to claim 9 wherein theerror concealment stage modifies the strength of the deblocking filterby modifying a boundary strength value on transitions between pairs ofconcealed macroblocks.
 12. The decoder according to claim 9 wherein theerror concealment stage modifies a Quantization Parameter (QP) averagefor the deblocking filter between concealed macroblocks and correctlyreceived macroblocks.
 13. The decoder according to claim 9 wherein theerror concealment stage modifies a Quantization Parameter (QP) averagefor the deblocking filter between pairs of concealed macroblocks. 14.The decoder according to claim 10 wherein the error concealment stagemodifies a Quantization Parameter (QP) average for the deblocking filterbetween concealed macroblocks and correctly received macroblocks. 15.The decoder according to claim 10 wherein the error concealment stagemodifies a Quantization Parameter (QP) average for the deblocking filterbetween pairs of concealed macroblocks.
 16. A method for smoothingtransitions in decoded macroblocks, comprising the steps of: detectingif a decoded macroblock has errors attributable to missing/corruptedpixel values, and if so, concealing errors by estimating themissing/corrupted pixel values from previously transmitted macroblocksto yield an error-concealed macroblock; and filtering the errorconcealed-macroblock by a deblocking filter to smooth the transitionsartificially created by the error concealment algorithm.
 17. The methodaccording to claim 16 further comprising the step of varying thestrength of the deblocking performed by the deblocking filter inaccordance with error concealment.
 18. The method according to claim 17wherein the step of modifying the strength of the deblocking filtercomprises modifying a boundary strength value on transitions betweenconcealed macroblocks and error-free (correctly received) macroblocks.19. The method according to claim 17 wherein the step of modifying thestrength of the deblocking filter comprises modifying a boundarystrength value on transitions between pairs of concealed macroblocks.20. The method according to claim 17 further comprising the step ofmodifying a Quantization Parameter (QP) average for the deblockingfilter between concealed macroblocks and correctly received macroblocks.21. The method according to claim 17 further comprising the step ofmodifying a Quantization Parameter (QP) average for the deblockingfilter between pairs of concealed macroblocks.
 22. The method accordingto claim 18 wherein the error concealment stage modifies a Parameter(QP) average for the deblocking filter between concealed macroblocks andcorrectly received macroblocks.
 23. The method according to claim 18further comprising the step of modifying a Quantization Parameter (QP)average for the deblocking filter between pairs of concealedmacroblocks.
 24. The method according to claim 17 further comprising thestep of varying each of a pair of offset values A and B for thedeblocking filter.